Thera Bank Personal Loan Campaign Data Description:¶
The dataset contains data on 5000 customers. The data include customer demographic information (age, income, etc.), the customer's relationship with the bank (mortgage, securities account, etc.), and the customer response to the last personal loan campaign (Personal Loan). Among these 5000 customers, only 480 (= 9.6%) accepted the personal loan that was offered to them in the earlier campaign.
Domain:¶
Banking
Context:¶
This case is about a bank (Thera Bank) whose management wants to explore ways of converting its liability customers to personal loan customers (while retaining them as depositors). A campaign that the bank ran last year for liability customers showed a healthy conversion rate of over 9% success. This has encouraged the retail marketing department to devise campaigns with better target marketing to increase the success ratio with a minimal budget.
Attribute Information:¶
ID: Customer ID Age: Customer's age in completed years Experience: #years of professional experience Income: Annual income of the customer ($000) ZIP Code: Home Address ZIP code. Family: Family size of the customer CCAvg: Avg. spending on credit cards per month ($000) Education: Education Level. 1: Undergrad; 2: Graduate; 3: Advanced/Professional Mortgage: Value of house mortgage if any. ($000) Personal Loan: Did this customer accept the personal loan offered in the last campaign? Securities Account: Does the customer have a securities account with the bank? CD Account: Does the customer have a certificate of deposit (CD) account with the bank? Online: Does the customer use internet banking facilities? Credit card: Does the customer use a credit card issued by the bank?
- Import the datasets and libraries, check datatype, statistical summary, shape, null values or incorrect imputation
- EDA: Study the data distribution in each attribute and target variable, share your findings
- Split the data into training and test set in the ratio of 70:30 respectively
- Use Logistic Regression model to predict the number of a customers buying personal loans. Print all the metrics related for evaluating the model performance
- Give your reasoning on how can the model perform better?
- Give Business understanding of your model
Import the datasets and libraries, check datatype, statistical summary, shape, null values or incorrect imputation.¶
In [1]:
import warnings
warnings.filterwarnings('ignore')
import libraries¶
In [2]:
# data read and structuring
import pandas as pd
import numpy as np
# visualization
import seaborn as sns
import matplotlib.pyplot as plt
# profiling
import pandas_profiling
# model building
from sklearn.linear_model import LogisticRegression
# data preparing
from sklearn.model_selection import train_test_split
# check error values
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
# custom display output
from IPython.display import display, HTML
import scipy.stats as stats
# scalars to normalize and impute
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import PowerTransformer
from sklearn.preprocessing import QuantileTransformer
from sklearn.preprocessing import RobustScaler
%matplotlib inline
sns.set(color_codes=True)
get the data¶
In [3]:
data = pd.read_csv('Bank_Personal_Loan_Modelling.csv')
In [4]:
# quick peak of the data
data.head()
Out[4]:
In [5]:
data.describe().transpose()
# from the data 5 point summary below,
# - we can ignore features 'ID' and 'ZIP Code'. The 5-point summary do not make sense, neither are they the independent features
# - 'Securities Account', 'CD Account', 'Online' and 'CreditCard' are discrete binary categorical value and 5 -point summary does not apply but are independent features
# - 'Education' is multi-class categorical feature that may influence the outcome
# Contineous features that impact the outcomes are 'Age', 'Experience', 'Income', 'Family', 'CCAvg', 'Mortgage'
# target or dependent or outcome variable is 'Personal Loan'. Since this is a discrete binary categorical value, we will need to apply 'Logistics Regression'
# From this data we can also see that min value of experience is ''-3' something that we will need to correct using imputation
# no need for ‘ID’, ‘ZIP_Code’ columns for further analysis since ‘ID’ and ‘ZIP_Code’ are just numbers of series
Out[5]:
from the data 5 point summary below,¶
we can ignore features 'ID' and 'ZIP Code'. The 5-point summary do not make sense, neither are they the independent features
'Securities Account', 'CD Account', 'Online' and 'CreditCard' are discrete binary categorical value and 5 -point summary does not apply but are independent features that will influence the outcome (Personal Loan)
'Education' is multi-class categorical feature that may influence the outcome
Contineous features that impact the outcomes are 'Age', 'Experience', 'Income', 'Family', 'CCAvg', 'Mortgage'
target or dependent or outcome variable is 'Personal Loan'. Since this is a discrete binary categorical value, we will apply 'Logistics Regression'
From this data we can also see that min value of experience is ''-3' something that we will need to correct using imputation
no need for ‘ID’, ‘ZIP_Code’ columns for further analysis since ‘ID’ and ‘ZIP_Code’ are just numbers of series
In [6]:
data.shape
Out[6]:
In [7]:
data.dtypes
Out[7]:
In [8]:
# check null or missing values
data.isnull().sum()
Out[8]:
The data set got 0 missing cells.¶
In [9]:
data.info()
5 point summary informed of incorrect/negative 'Experience' value. We will correct it and use the median in the given age range¶
In [10]:
data[data.Experience < 0].Experience.count()
Out[10]:
EDA: Study the data distribution in each attribute and target variable, share your findings¶
Lets check the correlation between feature and drop features that have strong correlation¶
In [11]:
data.corr()
Out[11]:
In [12]:
plt.figure(figsize=(20,15))
sns.heatmap(data.corr(), annot=True);
profile the data¶
In [15]:
data.profile_report()
Out[15]:
observation with the 5 -point summary and correlation¶
- Mortgage is negatively skewed but we will allow the algorithm to manage it and not impute or correct anything in it
- CCAvg is negatively skewed but we will allow the algorithm to manage it and not impute or correct anything in it
- No duplicate rows
Number of unique in each column¶
In [16]:
data.nunique()
Out[16]:
Observation with the number of unique values¶
- We have a good spread of Age, Experience, Income, Family Size, Mortgage
- data with 467 zip code informs that the data was collected from multiple communities
Number of people with zero mortgage¶
In [17]:
data[data.Mortgage == 0].Mortgage.count()
Out[17]:
Out of 5000 instances, 3462 have no current Mortgages¶
Number of people with zero credit card spending per month¶
In [18]:
data[data.CCAvg == 0].CCAvg.count()
Out[18]:
Out of 5000 instances, 106 have 0 average credit card spending¶
Value counts of all categorical columns.¶
In [19]:
data.columns
categorical_columns = ['Family', 'Education', 'Personal Loan', 'Securities Account',
'CD Account', 'Online', 'CreditCard']
for col in categorical_columns:
print(data[col].value_counts())
print()
Percent of all categorical columns.¶
In [20]:
for col in categorical_columns:
print(data[col].value_counts(normalize=True) * 100)
print()
42% of the candidates are graduated, while 30% are professional and 28% are Undergraduate.¶
Family size is fairly distributed¶
Approximately 94% of the customer doesn’t have a certificate of deposit (CD) account with the bank.¶
Around 71% of the customer doesn’t use a credit card issued by Bank.¶
Around 60% of customers use internet banking facilities.¶
Around 90% of the customer doesn’t accept the personal loan offered in the last campaign.¶
Around 90% of the customer doesn’t have a securities account with the bank.¶
Univariate and Bivariate¶
In [21]:
# Personal loan count based on educational level
def independent_count_plot(independent):
plt.figure(figsize=(10,7))
plt.title('Count of ' + independent + ' feature', fontsize=20)
plt.xlabel(independent, fontsize=10)
plt.ylabel('Count', fontsize=10)
sns.countplot(x=independent, data=data)
plt.show()
# count of a given independent feature against the dependent feature
def count_plot(independent, dependent):
indep_vs_dep = pd.crosstab(data[independent], data[dependent])
print (indep_vs_dep)
print()
print (indep_vs_dep.div(indep_vs_dep.sum(1).astype(float), axis = 0) * 100)
print()
plt.figure(figsize=(10,7))
plt.title(dependent + ' count per ' + independent + ' feature', fontsize=20)
sns.countplot(x=dependent, hue=independent, data=data);
plt.show()
indep_vs_dep.div(indep_vs_dep.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
def univariant_categorical(independent, dependent):
count_plot(independent, dependent)
independent_count_plot(independent)
# Univariant: Personal loan count based on educational level
categorical_columns = ['Family', 'Education', 'Securities Account', 'CD Account', 'Online', 'CreditCard']
for col in categorical_columns:
univariant_categorical(col, 'Personal Loan')
Personal Loan is highly correlated with Income, average spending on Credit cards, mortgage & if the customer has a certificate of deposit (CD) account with the bank.¶
In [22]:
def box_independent(independent):
plt.figure(figsize=(10,7))
sns.boxplot(data=data, x = independent);
plt.show()
def dist_plot(independent):
plt.figure(figsize=(10,7))
sns.distplot(data[independent]);
def univariant_contineous(independent, dependent):
box_independent(independent)
dist_plot(independent)
contineous_feature = ['Age', 'Experience', 'Income', 'CCAvg', 'Mortgage']
for col in contineous_feature:
univariant_contineous(col, 'Personal Loan')
Observation from the boxplot and histogram¶
- Age and Experience has got a good distribution
- Income and CCAvg is positively skewed
- Mortgage is highly skewed to the right and probably can be ignored when fitting the model.
In [23]:
sns.pairplot(data[contineous_feature]);
Observation from the scatter plots¶
- Age and Experience are highly correlated as was also observed in the correlation (using heatmap and corr())
- Income have a correlation have a relation where with the increase in income, customer's Mortgage and CCAvg also increases
- Mortgage, Age, Experience do not have linear relation and is scattered
In [24]:
# average_contineous_plot(independent, dependent)
def average_contineous_plot(independent, dependent):
data.groupby(dependent)[independent].mean().plot(kind='bar')
plt.title('Average ' + independent + ' effect on ' + dependent, fontsize=20)
plt.show()
print()
for col in contineous_feature:
average_contineous_plot(col, 'Personal Loan')
Observation from the chart above¶
- Average Age with Personal loan is almost the same, which implies that it is not providing a good relation
- People with high income buys more personal loan
- People with high CCAvg spending buys more personal loan
- People having more average mortgage buys more personal loan
In [25]:
# Let's look at the skewness of data
skew_df = pd.DataFrame({'Skewness' : [stats.skew(data.Age),
stats.skew(data.Experience),
stats.skew(data.Income),
stats.skew(data.CCAvg)
,stats.skew(data.Mortgage)]},
index=['Age','Experience','Income','CCAvg','Mortgage'])
skew_df
Out[25]:
In [26]:
plt.figure(figsize=(20,30))
plt.subplot(6,2,1)
sns.scatterplot(data.Age, data.Experience, hue = data['Personal Loan']);
plt.subplot(6,2,2)
sns.scatterplot(data.Family, data.Income, hue = data['Personal Loan']);
plt.subplot(6,2,3)
sns.scatterplot(data.Income, data.Mortgage, hue = data['Personal Loan']);
plt.subplot(6,2,4)
sns.scatterplot(data.CCAvg, data.Income, hue = data['Personal Loan'])
plt.subplot(6,2,5)
sns.scatterplot(data.Education, data.Income, hue = data['Personal Loan']);
plt.subplot(6,2,6)
sns.scatterplot(data.Income, data['ZIP Code'], hue = data['Personal Loan']);
plt.figure(figsize=(10,7));
plt.show()
From all of the the above plots, we can infer that¶
- customers who are more educated have a higher probability of buying personal loans.
- The number of family members not significantly affect buying of personal loan.
- The customer who has a certificate of deposit (CD) account with the bank seems to buy personal loans from the bank.
- The customer who uses or doesn’t use a credit card issued by Bank doesn’t seem to affect the probability of buying a personal loan.
- The customer who uses or doesn’t use internet banking facilities seems to not affect the probability of buying personal loans.
- The customers who have or don’t have securities account with the bank do not affect the probability of buying a personal loan.
Get data model ready¶
- Remove negative experience
- Replace all 0s with the mean for 'Age', 'Family'. Note that 0 is a valid value for 'Experience', 'Income', 'CCAvg', 'Mortgage' and all other categorical variables
- Perform one-hot encoding on the multi-class categorical features
- Treat outliers using scalar
- Remove 'ID', 'ZIP code' from the data
- Age is highly correlated to the Experience, we can skip the Age and use only the experience
Remove negative experience¶
In [27]:
data_processed = data.copy()
In [28]:
# 5 point summary informed of incorrect/negative 'Experience' value. We need to correct it and use the
# median in the given age range
# Binning
bin_edges = [22, 27, 32, 37, 42, 47, 52, 57, 62, 68] # edges to define intervals
bin_labels = ['22-27', '28-32', '33-37','38-42', '43-47', '48-52',
'53-57','58-62', '63-68'] # labels to denote each interval
data['AgeBin'] = pd.cut(data_processed['Age'], bins=bin_edges , labels=bin_labels)
# pd.cut is used to divide the continous column in different groups as per bin egges and named according to bin label.
data_processed[data_processed.Experience < 0]['Experience'].count()
Out[28]:
In [29]:
#
# Correct the experience value with the average experience for a given age group
#
data.groupby('AgeBin')['Experience'].mean().apply(np.ceil)['22-27']
avgExp = data[data.Experience >= 0].groupby('AgeBin')['Experience'].mean().apply(np.ceil)
avgExp
for i, j in data[data.Experience < 0].iterrows():
data.at[i,'Experience'] = avgExp[j['AgeBin']]
data.drop('AgeBin', axis=1, inplace=True)
data[data.Experience < 0]
Out[29]:
Experience now is been taken care with the average experience in the given age group¶
Replace all 0s with the mean for 'Age', 'Family'. Note that 0 is a valid value for 'Experience', 'Income', 'CCAvg', 'Mortgage' and all other categorical variables¶
In [30]:
# Run imputer to fill all places where 0 is not a valid value with its mean
from sklearn.impute import SimpleImputer
rep_0 = SimpleImputer(missing_values=0, strategy="mean")
cols=['Age','Family']
imputer = rep_0.fit(data_processed[cols])
data_processed[cols] = imputer.transform(data_processed[cols])
data_processed.head(10)
Out[30]:
In [31]:
data_processed.info()
In [32]:
data_processed.head()
Out[32]:
In [33]:
# convert Education and Family to categorical data and apply one hot encoding
data_processed['Education'] = data_processed['Education'].astype('category')
data_processed['Family'] = data_processed['Family'].astype('category')
In [35]:
data_processed.info()
In [36]:
data_processed = data_processed.drop(columns=['ID', 'ZIP Code', 'Age'])
Treat the outliers using RobustScalar¶
In [37]:
# treat the outliers using RobustScalar
from sklearn import preprocessing
scaler = preprocessing.RobustScaler()
# Get list of independent variables to scale
variables_to_scale = ['Income', 'CCAvg', 'Mortgage', 'Experience']
data_processed[variables_to_scale] = scaler.fit_transform(data_processed[variables_to_scale])
data_processed
Out[37]:
apply one-hot encoding on the multi-class categorical variables¶
In [38]:
data_processed = pd.get_dummies(data_processed, drop_first=True)
data_processed
Out[38]:
We did not removed the 'Mortgage' from the independent feature since it had a very high effect on the 'Personal loan'¶
Split the data into training and test set in the ratio of 70:30 respectively¶
In [39]:
from sklearn.model_selection import train_test_split
pdata = data_processed.copy()
pdata
X = pdata.drop('Personal Loan',axis=1) # Predictor feature columns (8 X m)
y = pdata['Personal Loan'] # Predicted class (1=True, 0=False) (1 X m)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
X_train.head()
Out[39]:
Lets check the split data
In [40]:
print("{0:0.2f}% data is in training set".format((len(X_train)/len(pdata.index)) * 100))
print("{0:0.2f}% data is in test set".format((len(X_test)/len(pdata.index)) * 100))
In [41]:
print("Original Personal Loan True Values : {0} ({1:0.2f}%)".format(len(pdata.loc[pdata['Personal Loan'] == 1]), (len(pdata.loc[pdata['Personal Loan'] == 1])/len(pdata.index)) * 100))
print("Original Personal Loan False Values : {0} ({1:0.2f}%)".format(len(pdata.loc[pdata['Personal Loan'] == 0]), (len(pdata.loc[pdata['Personal Loan'] == 0])/len(pdata.index)) * 100))
print("")
print("Training Personal Loan True Values : {0} ({1:0.2f}%)".format(len(y_train[y_train[:] == 1]), (len(y_train[y_train[:] == 1])/len(y_train)) * 100))
print("Training Personal Loan False Values : {0} ({1:0.2f}%)".format(len(y_train[y_train[:] == 0]), (len(y_train[y_train[:] == 0])/len(y_train)) * 100))
print("")
print("Test Personal Loan True Values : {0} ({1:0.2f}%)".format(len(y_test[y_test[:] == 1]), (len(y_test[y_test[:] == 1])/len(y_test)) * 100))
print("Test Personal Loan False Values : {0} ({1:0.2f}%)".format(len(y_test[y_test[:] == 0]), (len(y_test[y_test[:] == 0])/len(y_test)) * 100))
print("")
Use Logistic Regression model to predict the number of a customers buying personal loans. Print all the metrics related for evaluating the model performance¶
Logistics Regression using statsmodel¶
In [42]:
#Build the logistic regression model
import statsmodels.api as sm
logit = sm.Logit(y_train, sm.add_constant(X_train))
lg = logit.fit()
In [43]:
#Summary of logistic regression
from scipy import stats
stats.chisqprob = lambda chisq, df: stats.chi2.sf(chisq, df)
print(lg.summary())
Interpretation of Pseudo R^2¶
A pseudo R^2 of 65% indicates that 65% of the uncertainty of the intercept only model is explained by the full model
In [44]:
#Calculate Odds Ratio, probability
##create a data frame to collate Odds ratio, probability and p-value of the coef
lgcoef = pd.DataFrame(lg.params, columns=['coef'])
lgcoef.loc[:, "Odds_ratio"] = np.exp(lgcoef.coef)
lgcoef['probability'] = lgcoef['Odds_ratio']/(1+lgcoef['Odds_ratio'])
lgcoef['pval']=lg.pvalues
pd.options.display.float_format = '{:.2f}'.format
In [45]:
# FIlter by significant p-value (pval <0.1) and sort descending by Odds ratio
lgcoef = lgcoef.sort_values(by="Odds_ratio", ascending=False)
pval_filter = lgcoef['pval']<=0.1
lgcoef[pval_filter]
Out[45]:
- Customer with high education have very high probablility (approx 98% to 99%) of buying Personal Loan
- Customer holding CD Account have %97 chances of buying the Personal Loan
- Family of all sizes have probability of buy Personal Loan
Model building using Logistic Regression from Sklearn¶
We will use the sklearn library to build the model and make predictions
In [46]:
from sklearn import metrics
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, recall_score, precision_score, f1_score, roc_auc_score,accuracy_score, classification_report
# Fit the model on train
model = LogisticRegression(random_state=42)
model.fit(X_train, y_train)
Out[46]:
In [47]:
y_predict = model.predict(X_test)
In [48]:
## function to get confusion matrix in a proper format
def draw_cm( actual, predicted):
cm = confusion_matrix( actual, predicted)
sns.heatmap(cm, annot=True, fmt='.2f', xticklabels = [0,1] , yticklabels = [0,1] )
plt.ylabel('Observed')
plt.xlabel('Predicted')
plt.show()
In [49]:
print("Training accuracy",model.score(X_train,y_train))
print()
print("Testing accuracy",model.score(X_test, y_test))
print()
print('Confusion Matrix')
print(draw_cm(y_test,y_predict))
print()
print("Recall:",recall_score(y_test,y_predict))
print()
print("Precision:",precision_score(y_test,y_predict))
print()
print("F1 Score:",f1_score(y_test,y_predict))
print()
print("Roc Auc Score:",roc_auc_score(y_test,y_predict))
The confusion matrix
True Positives (TP): we correctly predicted that they will take personal loans 109
True Negatives (TN): we correctly predicted that they will not take personal loan 1334
False Positives (FP): we incorrectly predicted that they will take personal loan (a "Type I error") 9 Falsely predict positive Type I error
False Negatives (FN): we incorrectly predicted that they will not take personal loans (a "Type II error") 48 Falsely predict negative Type II error
In [50]:
#AUC ROC curve
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
logit_roc_auc = roc_auc_score(y_test, model.predict(X_test))
fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])
plt.figure()
plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)' % logit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('Log_ROC')
plt.show()
In [51]:
print(classification_report(y_test, y_predict))
Give your reasoning on how can the model perform better?¶
Check model output and change model parameters¶
And here we are with a classification report which consists of a precision, recall, f1-score & support, Also the accuracy_score and a 2*2 confusion matrix. How to determine if our model has done well…? Well first have a look at the accuracy, 96% accuracy is not a small thing, but we know from the data that the number of buyer’s percentage to the non-buyer percentage is very less. Hence accuracy didn’t play a big role in determining how our model performed.
We will look upon the error types now:¶
- True Positives (TP): we correctly predicted that they will take personal loans 109
- True Negatives (TN): we correctly predicted that they will not take personal loan 1334
- False Positives (FP): we incorrectly predicted that they will take personal loan (a "Type I error") 11 Falsely predict positive Type I error
- False Negatives (FN): we incorrectly predicted that they will not take personal loans (a "Type II error") 48 Falsely predict negative Type II error -- In our case, the null hypothesis presents the non-buyer case, while the alternate hypothesis is the buyer one.
We must concentrate upon or reduce type II error here since we are interested in the customers who had actually bought personal loans, but our model predicted them to be a non-buyer.
Check False Negatives¶
Eventually, we can concentrate upon our confusion matrix and look for the False Negatives which in this case is 48, less the number of False Negatives, wiser our model will be or we can directly look upon the recall for ‘1’ which in this case is 69%. So, in this case, out of the total number of customers who actually bought personal loans our model is only able to pick 69% of customers of them to be correctly predicted.
We will also compare the area under roc curve to determine how our model performs. In this case, the area comes out to be around 84% which is good.¶
In [52]:
# Checking Parameters of logistic regression
model.get_params()
#If we dont specify the parameters in the model it takes default value
# We can change the solver, penalty, C, multi_class, fit-intercept, dual, cweight and find the effect on the
# accuracy, recall, precision, roc, auc
Out[52]:
Fit the train data with different parameters of Logistic Regression and pick the top best models that had good accuracy score, recall score, precision score and roc score¶
- We will run through all parameter combinations in the cell below and prepare a data frame with the top performers
In [53]:
# 'penalty': 'l1', 'l2', 'elasticnet' or 'none',
### 'newton-cg', 'sag' and 'lbfgs' solvers support only l2 penalties
### 'elasticnet' is only supported by the 'saga' solver
###
### multi_class : str, {'ovr', 'multinomial', 'auto'}
### - If the option chosen is 'ovr', then a binary problem is fit for each
### label. For 'multinomial' the loss minimised is the multinomial loss fit
### across the entire probability distribution, *even when the data is
### binary*. 'multinomial' is unavailable when solver='liblinear'.
### 'auto' selects 'ovr' if the data is binary, or if solver='liblinear',
### and otherwise selects 'multinomial'.
###
### dual: True or False
### - bool, optional (default=False)
### Dual or primal formulation. Dual formulation is only implemented for
### l2 penalty with liblinear solver. Prefer dual=False when
### n_samples > n_features.
###
### fit_intercept: True or False
###
### solver: {'newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'}
###
cols = ['solver', 'penalty', 'multi_class', 'dual', 'fit_intercept',
'class_weight', 'C',
'train score', 'test score', 'recall score', 'precision score',
'roc auc score']
col_types = [('solver', str),
('penalty', str),
('multi_class', str),
('dual', bool),
('fit_intercept', bool),
('class_weight', str), ('C', np.float128),
('train score', np.float128), ('test score', np.float128),
('recall score', np.float128),
('precision score', np.float128),
('roc auc score', np.float128)]
df_compare_model = pd.DataFrame({k: pd.Series(dtype=t) for k, t in col_types})
idx = 0
rows_list = []
def regression_using_params():
test_accuracy = 0.0
train_accuracy = 0.0
test_rscore = 0.0
test_pscore = 0.0
test_auc = 0.0
global df_compare_model
global idx
global rows_list
penalties = ['l1', 'l2', 'elasticnet', 'none']
multi_class = ['ovr', 'multinomial', 'auto']
dual = [True, False]
fit_intercept = [True, False]
solver = ['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga']
C = [0.010000,0.100000,0.250000,0.500000,0.750000,1.000000]
cweight = ['balanced', None]
for solv in solver:
for p in penalties:
for mclass in multi_class:
for d in dual:
for fi in fit_intercept:
for cw in cweight:
for cval in C:
lr = LogisticRegression(solver=solv,
max_iter=1000,
penalty=p,
dual=d,
fit_intercept=fi,
class_weight=cw,
C=cval,
random_state=42)
# Solver newton-cg supports only 'l2' or 'none' penalties.
if solv == 'newton-cg' and p not in ('l2', 'none'):
continue
# Solver newton-cg supports only dual=False
if solv == 'newton-cg' and d == True:
continue
# Solver lbfgs supports only 'l2' or 'none' penalties.
if solv == 'lbfgs' and p not in ('l2', 'none'):
continue
# Solver lbfgs supports only dual=False
if solv == 'lbfgs' and d == True:
continue
# The combination of penalty='l1' and loss='logistic_regression' are not supported
# when dual=True, Parameters: penalty='l1', loss='logistic_regression', dual=True
if p == 'l1' and d == True:
continue
# Only 'saga' solver supports elasticnet penalty
if solv != 'saga' and p == 'elasticnet':
continue
# penalty='none' is not supported for the liblinear solver
if solv == 'liblinear' and p == 'none':
continue
# Solver sag supports only 'l2' or 'none' penalties
if solv == 'sag' and p not in ('l2', 'none'):
continue
# Solver sag supports only dual=False
if solv == 'sag' and d == True:
continue
# Solver saga supports only dual=False
if solv == 'saga' and d == True:
continue
# l1_ratio must be between 0 and 1; got (l1_ratio=None)
# Liner regression on: saga elasticnet False True
if solv == 'saga' and p == 'elasticnet' and d == False:
continue
lr.fit(X_train, y_train)
model_score_train = lr.score(X_train, y_train)
model_score_test = lr.score(X_test, y_test)
y_predict = model.predict(X_test)
rscore = recall_score(y_test,y_predict)
pscore = precision_score(y_test,y_predict)
f1 = f1_score(y_test,y_predict)
ras = roc_auc_score(y_test,y_predict)
new_row = {'solver':solv, 'penalty': p, 'multi_class': mclass, 'dual': d,
'fit_intercept': fi,
'class_weight': cw, 'C':cval,
'train score': model_score_train,
'test score': model_score_test,
'recall score': rscore,
'precision score': pscore,
'roc auc score': ras}
#
# pick models that have high accuracy
#
if (model_score_test >= test_accuracy and
model_score_train >= train_accuracy):
# print (new_row)
test_accuracy = model_score_test
train_accuracy = model_score_train
test_rscore = rscore
test_pscore = pscore
test_auc = ras
df_compare_model = df_compare_model.append([new_row], ignore_index=True)
# rows_list.append(new_row)
# idx = idx + 1
# df_compare_model = df_compare_model.append([new_row], ignore_index=True)
regression_using_params()
# df_compare_model
In [54]:
# find the top performer out of all the permutation and combinations of the regression parameter
df_compare_model.sort_values(['test score', 'train score', 'recall score', 'precision score', 'roc auc score'],
ascending=[False, False, False, False, False]).head(50)
Out[54]:
there are multiple parameter combinations that provide almost the same accuracy, precision, recall and auc. We picked one the top one after sorting on these scores.¶
In [55]:
## Best testing accuracy is obtained with multiple parameter combition, we will pick one of them.
# 'solver': newton-cg,
# 'penalty': none,
# 'multi_class': ovr
# 'dual': False,
# 'fit_intercept': True,
# 'class_weight': None,
# 'C' {1.00}
# We will pick one of the combination
In [56]:
#Therefore final selected model is
model = LogisticRegression(random_state=42, solver='newton-cg', multi_class='ovr', dual=False,
C=1, class_weight=None, fit_intercept=True)
model.fit(X_train, y_train)
y_predict = model.predict(X_test)
print("Trainig accuracy",model.score(X_train,y_train))
print()
print("Testing accuracy",model.score(X_test, y_test))
print()
print('Confusion Matrix')
print(draw_cm(y_test,y_predict))
print()
print("Recall:",recall_score(y_test,y_predict))
print()
print("Precision:",precision_score(y_test,y_predict))
print()
print("F1 Score:",f1_score(y_test,y_predict))
print()
print("Roc Auc Score:",roc_auc_score(y_test,y_predict))
In [57]:
# Additional
from yellowbrick.classifier import ClassificationReport, ROCAUC
# Visualize model performance with yellowbrick library
viz = ClassificationReport(model)
viz.fit(X_train, y_train)
viz.score(X_test, y_test)
viz.show()
roc = ROCAUC(model)
roc.fit(X_train, y_train)
roc.score(X_test, y_test)
roc.show()
Out[57]:
After iterating through the model parameters, we got a slightly better precision, F1 score and Roc Auc Score compared to the default¶
We observed that with the treated data i.e. one-hot encoding, treating outliers, correcting negative experience, replace 0 age and family size with the mean, most of the parameter combination including the default performed fairly close.¶
Give Business understanding of your model¶
Model understanding¶
And here we are with a classification report which consists of a precision, recall, f1-score & support, Also the accuracy_score and a 2*2 confusion matrix. How to determine if our model has done well…? Well first have a look at the accuracy, 96.2% accuracy is not a small thing, but we know from the data that the number of buyer’s percentage to the non-buyer percentage is very less. Hence accuracy didn’t play a big role in determining how our model performed. We got a small increase in the accuracy compared to the default parameter.¶
We will look upon the error types now:¶
- True Positives (TP): we correctly predicted that they will take personal loans 109
- True Negatives (TN): we correctly predicted that they will not take personal loan 1334
- False Positives (FP): we incorrectly predicted that they will take personal loan (a "Type I error") 9 Falsely predict positive Type I error
- False Negatives (FN): we incorrectly predicted that they will not take personal loans (a "Type II error") 48 Falsely predict negative Type II error -- In our case, the null hypothesis presents the non-buyer case, while the alternate hypothesis is the buyer one. We must concentrate upon or reduce type II error here since we are interested in the customers who had actually bought personal loans, but our model predicted them to be a non-buyer.
Eventually, we can concentrate upon our confusion matrix and look for the False Negatives which in this case is 48, less the number of False Negatives, wiser our model will be or we can directly look upon the recall for ‘1’ which in this case is 69.42%. So, in this case, out of the total number of customers who actually bought personal loans our model is only able to pick 69.42% of customers of them to be correctly predicted.¶
We will also compare the area under roc curve to determine how our model performs. In this case, the area comes out to be around 84.37% which is good.¶
Predicted that customer who are interested in buying personal loan will be reached and provided the loan
Here the bank wants to reach people who will take personal loan but our model predicted they will not take loan i.e. less number of False Negative, if FN is high, bank would lose on prospect customers. So that the bank doesn't lose money who are willing to take personal loans. Hence Recall is the important metric.
In case of False positive, bank will lose effort to reach out to a few people but that okay because the bank thought that these people will take personal loan but they did not take. This number is quite low i.e. 9 and the precision is quite high 92.3%
After achieving the desired accuracy we can deploy the model for practical use. As in the bank now can predict who is eligible for personal loan. They can use the model for upcoming customers.
Business understanding using EDA:¶
Personal Loan is highly correlated with Income, average spending on Credit cards, mortgage & if the customer has a certificate of deposit (CD) account with the bank.
The number of family members not significantly affect buying of personal loan.
The customer who has a certificate of deposit (CD) account with the bank seems to buy personal loans from the bank.
The customer who uses or doesn’t use a credit card issued by Bank doesn’t seem to affect the probability of buying a personal loan.
The customer who uses or doesn’t use internet banking facilities seems to not affect the probability of buying personal loans.
The customers who have or don’t have securities account with the bank do not affect the probability of buying a personal loan.
42% of the candidates are graduated, while 30% are professional and 28% are Undergraduate.
Family size is fairly distributed
Approximately 94% of the customer doesn’t have a certificate of deposit (CD) account with the bank.
Around 71% of the customer doesn’t use a credit card issued by UniversalBank.
Around 60% of customers use internet banking facilities.
Around 90% of the customer doesn’t accept the personal loan offered in the last campaign.
Around 90% of the customer doesn’t have a securities account with the bank.
We can target customers that are highly educated, high income, hold credit card, have mortgage account, holds CD account for personal loan¶
In [ ]: